{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![LOGO](../../../img/MODIN_ver2_hrz.png)\n",
    "\n",
    "<center><h2>Scale your pandas workflows by changing one line of code</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3: Not Implemented\n",
    "\n",
    "**GOAL**: Learn what happens when a function is not yet supported in Modin as well as how to extend Modin's functionality using the DataFrame Algebra."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When functionality has not yet been implemented, we default to pandas\n",
    "\n",
    "![](../../../img/convert_to_pandas.png)\n",
    "\n",
    "We convert a Modin dataframe to pandas to do the operation, then convert it back once it is finished. These operations will have a high overhead due to the communication involved and will take longer than pandas.\n",
    "\n",
    "When this is happening, a warning will be given to the user to inform them that this operation will take longer than usual. For example, `DataFrame.mask` is not yet implemented. In this case, when a user tries to use it, they will see this warning:\n",
    "\n",
    "```\n",
    "UserWarning: `DataFrame.mask` defaulting to pandas implementation.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concept for exercise: Default to pandas\n",
    "\n",
    "In this section of the exercise we will see first-hand how the runtime is affected by operations that are not implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import modin.pandas as pd\n",
    "import pandas\n",
    "import numpy as np\n",
    "import time\n",
    "import modin.config as cfg\n",
    "cfg.Engine.put(\"dask\")\n",
    "\n",
    "frame_data = np.random.randint(0, 100, size=(2**18, 2**8))\n",
    "df = pd.DataFrame(frame_data).add_prefix(\"col\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pandas_df = pandas.DataFrame(frame_data).add_prefix(\"col\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "modin_start = time.time()\n",
    "\n",
    "print(df.mask(df < 50))\n",
    "\n",
    "modin_end = time.time()\n",
    "print(\"Modin mask took {} seconds.\".format(round(modin_end - modin_start, 4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pandas_start = time.time()\n",
    "\n",
    "print(pandas_df.mask(pandas_df < 50))\n",
    "\n",
    "pandas_end = time.time()\n",
    "print(\"pandas mask took {} seconds.\".format(round(pandas_end - pandas_start, 4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concept for exercise: Register custom functions\n",
    "\n",
    "Modin's user-facing API is pandas, but it is possible that we do not yet support your favorite or most-needed functionalities. Your user-defined function may also be able to be executed more efficiently if you pre-define the type of function it is (e.g. map, reduce, etc.) using the DataFrame Algebra. To solve either case, it is possible to register a custom function to be applied to your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Registering a custom function for all query compilers\n",
    "\n",
    "To register a custom function for a query compiler, we first need to import it:\n",
    "\n",
    "```python\n",
    "from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n",
    "```\n",
    "\n",
    "The `PandasQueryCompiler` is responsible for defining and compiling the queries that can be operated on by Modin, and is specific to the pandas storage format. Any queries defined here must also both be compatible with and result in a `pandas.DataFrame`. Many functionalities are very simply implemented, as you can see in the current code: [Link](https://github.com/modin-project/modin/blob/7a8158873e77cb5f1a5a3b89be4ddac89f576269/modin/core/storage_formats/pandas/query_compiler.py#L216).\n",
    "\n",
    "If we want to register a new function, we need to understand what kind of function it is. In our example, we will try to implement a `kurtosis` on the unary negation of the values in the dataframe, which is a map (unargy negation of each cell) followed by a reduce. So we next want to import the function type so we can use it in our definition:\n",
    "\n",
    "```python\n",
    "from modin.core.dataframe.algebra import TreeReduce\n",
    "```\n",
    "\n",
    "Then we can just use the `TreeReduce.register` `classmethod` and assign it to the `PandasQueryCompiler`:\n",
    "\n",
    "```python\n",
    "PandasQueryCompiler.neg_kurtosis = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value, pandas.DataFrame.kurtosis)\n",
    "```\n",
    "\n",
    "We include `**kwargs` to the `lambda` function since the query compiler will pass all keyword arguments to both the map and reduce functions.\n",
    "\n",
    "Finally, we want a handle to it from the `DataFrame`, so we need to create a way to do that:\n",
    "\n",
    "```python\n",
    "def neg_kurtosis_func(self, **kwargs):\n",
    "    # The constructor allows you to pass in a query compiler as a keyword argument\n",
    "    return self.__constructor__(query_compiler=self._query_compiler.neg_kurtosis(**kwargs))\n",
    "\n",
    "pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\n",
    "```\n",
    "\n",
    "And then you can use it like you usually would:\n",
    "\n",
    "```python\n",
    "df.neg_kurtosis_custom()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n",
    "from modin.core.dataframe.algebra import TreeReduce"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PandasQueryCompiler.neg_kurtosis_custom = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value,\n",
    "                                                             pandas.DataFrame.kurtosis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pandas._libs import lib\n",
    "# The function signature came from the pandas documentation:\n",
    "# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurtosis.html\n",
    "def neg_kurtosis_func(self, axis=lib.no_default, skipna=True, level=None, numeric_only=None, **kwargs):\n",
    "    # We need to specify the axis for the query compiler\n",
    "    if axis in [None, lib.no_default]:\n",
    "        axis = 0\n",
    "    # The constructor allows you to pass in a query compiler as a keyword argument\n",
    "    # Reduce dimension is used for reduces\n",
    "    # We also pass all keyword arguments here to ensure correctness\n",
    "    return self._reduce_dimension(\n",
    "        self._query_compiler.neg_kurtosis_custom(\n",
    "            axis=axis, skipna=skipna, level=level, numeric_only=numeric_only, **kwargs\n",
    "        )\n",
    "    )\n",
    "\n",
    "pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Speed improvements\n",
    "If we were to try and replicate this functionality using the pandas API, we would need to call `df.applymap` with our unary negation function, and subsequently `df.kurtosis` on the result of the first call. Let's see how this compares with our new, custom function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "\n",
    "print(pandas_df.applymap(lambda cell_value: ~cell_value).kurtosis())\n",
    "\n",
    "end = time.time()\n",
    "pandas_duration = end - start\n",
    "print(\"pandas unary negation kurtosis took {} seconds.\".format(pandas_duration))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "\n",
    "print(df.applymap(lambda x: ~x).kurtosis())\n",
    "\n",
    "end = time.time()\n",
    "modin_duration = end - start\n",
    "print(\"Modin unary negation kurtosis took {} seconds.\".format(modin_duration))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_start = time.time()\n",
    "\n",
    "print(df.neg_kurtosis_custom())\n",
    "\n",
    "custom_end = time.time()\n",
    "modin_custom_duration = custom_end - custom_start\n",
    "print(\"Modin neg_kurtosis_custom took {} seconds.\".format(modin_custom_duration))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display\n",
    "\n",
    "display(Markdown(\"### As expected, Modin is {}x faster than pandas when chaining the functions; however we see that our custom function is even faster than that - beating pandas by {}x, and Modin (when chaining the functions) by {}x!\".format(round(pandas_duration / modin_duration, 2), round(pandas_duration / modin_custom_duration, 2), round(modin_duration / modin_custom_duration, 2))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations! You have just implemented new DataFrame functionality!\n",
    "\n",
    "## Consider opening a pull request: https://github.com/modin-project/modin/pulls\n",
    "\n",
    "For a complete list of what is implemented, see the [Supported APIs](https://modin.readthedocs.io/en/latest/supported_apis/index.html) section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test your knowledge: Add a custom function for another tree reduce: finding `DataFrame.mad` after squaring all of the values\n",
    "\n",
    "See the pandas documentation for the correct signature: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mad.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "modin_mad_custom_start = time.time()\n",
    "\n",
    "# Implement your function here! Put the result of your custom squared `mad` in the variable `modin_mad_custom`\n",
    "# Hint: Look at the kurtosis walkthrough above\n",
    "\n",
    "modin_mad_custom = ...\n",
    "print(modin_mad_custom)\n",
    "\n",
    "modin_mad_custom_end = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluation code, do not change!\n",
    "modin_mad_start = time.time()\n",
    "modin_mad = df.applymap(lambda x: x**2).mad()\n",
    "print(modin_mad)\n",
    "modin_mad_end = time.time()\n",
    "\n",
    "assert modin_mad_end - modin_mad_start > modin_mad_custom_end - modin_mad_custom_start, \\\n",
    "    \"Your implementation was too slow, or you used the chaining functions approach. Try again\"\n",
    "assert modin_mad._to_pandas().equals(modin_mad_custom._to_pandas()), \"Your result did not match the result of chaining the functions, try again\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now that you are able to create custom functions, you know enough to contribute to Modin!\n",
    "\n",
    "**Please move on to [Exercise 4](./exercise_4.ipynb) when you are ready**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
